Skip to content

Ship the CUDA delegate in the wheel - #21645

Open
shoumikhin wants to merge 1 commit into
gh/shoumikhin/93/headfrom
gh/shoumikhin/95/head
Open

Ship the CUDA delegate in the wheel#21645
shoumikhin wants to merge 1 commit into
gh/shoumikhin/93/headfrom
gh/shoumikhin/95/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.

There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.

Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.

find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::backend_cuda)

A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:

executorch/lib/libexecutorch_backend_cuda.so     the delegate
executorch/lib/libexecutorch_extension_cuda.so   the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so   the GPU device code

Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.

The stream helper's header no longer includes cuda_runtime.h, which the wheel does not publish. It
only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.

Built a CUDA wheel, installed it into a clean environment, and:

  • ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the
    same weights and inputs (largest absolute difference 0).
  • built a C++ application against the installed wheel alone and ran the same model, matching the
    same reference.
  • confirmed one library defines the stream state and the GPU shims, not several. Extracting them
    into every consumer put three copies in one wheel, and a stream selected through one was invisible
    to the others.
  • confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
    that links the CUDA runtime has a relative path to it.
  • confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
  • a row that names a CUDA train is built with the CUDA option on rather than left to autodetection,
    so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
    wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
    installed cleanly and then reported the backend as unregistered when a model ran.
  • a row whose major CUDA version does not match the installed toolkit now fails the build. The
    declared packages and the loader paths come from the row while the binaries come from the
    toolkit, and nothing compared the two, so a cu126 row built against a 13.0 toolkit attached
    CUDA 12 metadata to binaries needing libcudart.so.13. An unrecognised train fails too, instead
    of silently reporting whatever the builder happened to have. The comparison is at major only, so a
    cu126 row built with a 12.8 toolkit still passes.
  • whether a row is a CUDA row is decided by asking if it names a supported train, rather than by
    listing the spellings that mean "no CUDA". Checked 16 row values including cpu-aarch64, rocm6.2
    and cu118; the previous list-based form was wrong on several, and each wrong answer made a
    non-CUDA wheel declare the CUDA runtime.
  • the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were
    optional unconditionally, so a wheel tagged +cu126 with no CUDA library at all passed every check.
  • the stream helper ships under either name it can be built with. The shim layer records it as a
    dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
    non-shared build shipped a shim whose dependency resolved to nothing.
  • the relative hops between shipped libraries are sized by how deep the library sits in the package.
    A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
    landed on a directory that does not exist, and the hop from lib/ climbed out of the package
    entirely, where an unrelated library with a matching soname could satisfy the dependency first.
  • the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an
    opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.

Also fixed in this commit:

  • The pre-build classifier now uses python3 rather than a bare python, and does not discard
    stderr. On a host without a python alias any CUDA row was silently rebuilt as a CPU row.
  • CU_VERSION=cpu pip install . is handled explicitly instead of running the CUDA-train parser
    over it, which previously turned cpu into pu through a character-set strip and reached the
    unsupported-train error.

Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.

Known gap, not introduced here: the Python Runtime.load_program path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.

@shoumikhin

shoumikhin commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@pytorch-bot

pytorch-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21645

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 3fa305d with merge base cc853ae (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2026
@github-actions github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Aug 7, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Aug 11, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: shoumikhin / name: shoumikhin (9007a7e)

The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.

There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.

Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.

```cmake
find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::backend_cuda)
```

A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:

```
executorch/lib/libexecutorch_backend_cuda.so     the delegate
executorch/lib/libexecutorch_extension_cuda.so   the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so   the GPU device code
```

Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.

The declared set includes the runtime compiler, because a shipped library links it to build kernels
at run time. Each declared package also needs its own directory recorded, since that is where the
loader looks. On the CUDA 12 packaging the compiler installs into its own directory, and omitting it
left that library unable to find the compiler even though the package was installed. The CUDA 13
packaging puts every component in one directory, so the same gap does not appear there.

The stream helper's header no longer includes `cuda_runtime.h`, which the wheel does not publish. It
only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.

Built a CUDA wheel, installed it into a clean environment, and:

- ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the
  same weights and inputs (largest absolute difference 0).
- built a C++ application against the installed wheel alone and ran the same model, matching the
  same reference.
- confirmed one library defines the stream state and the GPU shims, not several. Extracting them
  into every consumer put three copies in one wheel, and a stream selected through one was invisible
  to the others.
- confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
  that links the CUDA runtime has a relative path to it.
- confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
- a row that names a CUDA train is built with the CUDA option on rather than left to autodetection,
  so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
  wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
  installed cleanly and then reported the backend as unregistered when a model ran.
- a row whose major CUDA version does not match the installed toolkit now fails the build. The
  declared packages and the loader paths come from the row while the binaries come from the
  toolkit, and nothing compared the two, so a `cu126` row built against a 13.0 toolkit attached
  CUDA 12 metadata to binaries needing `libcudart.so.13`. An unrecognised train fails too, instead
  of silently reporting whatever the builder happened to have. Detection reads the toolkit major
  directly, so the guard fires on any mismatch rather than only on the three exact `(major, minor)`
  pairs the supported list carries; on those three pairs it behaved correctly before, and on every
  other minor it saw an empty detection and skipped the check.
- the row classifier and the packaging read the row the same way now, so both agree on what a row
  spelled with an unsupported minor means. The shell classifier reduces the row to digits and
  matches against `SUPPORTED_CUDA_VERSIONS`. Packaging did the same shape on the outer decision
  and then took only the first two digits when picking runtime packages, so `cu125` classified as
  CPU on one side and declared CUDA 12 on the other. Packaging now matches on the same digits and
  raises loudly on an unsupported train instead.
- whether a row is a CUDA row is decided by asking if it names a supported train, rather than by
  listing the spellings that mean "no CUDA". Checked 16 row values including `cpu-aarch64`, `rocm6.2`
  and `cu118`; the previous list-based form was wrong on several, and each wrong answer made a
  non-CUDA wheel declare the CUDA runtime.
- the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were
  optional unconditionally, so a wheel tagged `+cu126` with no CUDA library at all passed every check.
- the stream helper ships under either name it can be built with. The shim layer records it as a
  dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
  non-shared build shipped a shim whose dependency resolved to nothing.
- the relative hops between shipped libraries are sized by how deep the library sits in the package.
  A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
  landed on a directory that does not exist, and the hop from `lib/` climbed out of the package
  entirely, where an unrelated library with a matching soname could satisfy the dependency first.
- the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an
  opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.

Also fixed in this commit:

- The pre-build classifier resolves the Python interpreter (`python3` or `python`, whichever
  exists) instead of assuming one name, and no longer discards stderr. Builders disagree on the
  name: Linux and macOS provide `python3`, while the Windows builder runs inside a conda
  environment that provides only `python`. Assuming either name breaks the other platform, and
  treating the failure as "not a CUDA row" silently rebuilt a CUDA row as a CPU row.
- `CU_VERSION=cpu pip install .` is handled explicitly instead of running the CUDA-train parser
  over it, which previously turned `cpu` into `pu` through a character-set strip and reached the
  unsupported-train error.

Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.

Known gap, not introduced here: the Python `Runtime.load_program` path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.

ghstack-source-id: ff24a2d
ghstack-comment-id: 5219161655
Pull-Request: #21645
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant